Systems for implementing fallback behaviors for autonomous vehicles

ABSTRACT

Aspects of the disclosure relate to controlling a vehicle in an autonomous driving mode. The system includes a plurality of sensors configured to generate sensor data. The system also includes a first computing system configured to generate trajectories using the sensor data and send the generated trajectories to a second computing system. The second computing system is configured to cause the vehicle to follow a receive trajectory. The system also includes a third computing system configured to, when there is a failure of the first computer system, generate and send trajectories to the second computing system based on whether a vehicle is located on a highway or a surface street.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/180,267, filed Nov. 5, 2018, the entire disclosure of whichis incorporated by reference herein.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or items fromone location to another. Such vehicles may operate in a fully autonomousmode where passengers may provide some initial input, such as a pick upor destination location, and the vehicle maneuvers itself to thatlocation. While doing so, safety of passengers, cargo, and the vehicleis an important consideration. Accordingly, often these vehicles havefallback systems which essentially cause the vehicle to apply the brakesas hard and as quickly as possible in an emergency.

BRIEF SUMMARY

Aspects of the disclosure provide a system for controlling a vehicle inan autonomous driving mode. The system includes a plurality of sensorsconfigured to generate sensor data, a first computing system, and athird computing system. The first computing system is configured togenerate trajectories using the sensor data and send the generatedtrajectories to a second computing system configured to cause thevehicle to follow a received trajectory. The third computing system isconfigured to, when there is a failure of the first computer system,generate and send trajectories to the second computing system based on atype of road on which the vehicle is currently traveling.

In one example, the third computing system is further configured togenerate and send the trajectories based on whether a vehicle is locatedon a highway or a surface street. In another example, the system alsoincludes the second computing system. In another example, the failurerelates to one or more of the plurality of sensors, and the thirdcomputing system is configured to generate the trajectories furtherbased on which of the plurality of sensors is functioning. In anotherexample, the first computing system is configured to differentiatebetween different types of road users in the sensor data, and the secondcomputing system is not configured to differentiate between differenttypes of road users in the sensor data. In this example, the firstcomputing system is configured to generate different behaviorpredictions based on each of the different types of road users, whereinthe third computing system is configured to generate behaviorpredictions for all road users in the sensor data in a same way. In thisexample, the third computing system is configured to generate behaviorpredictions corresponding to either following a current lane or changinglanes in response to a detection of any given road user on a highway. Inanother example, the first computing system is configured to generatetrajectories according to a first list of possible maneuvers, and thethird computing system is configured to generate trajectories accordingto a second list of possible maneuvers that is smaller than the firstlist of possible maneuvers. In this example, the third computing systemis configured to determine the second list of possible maneuvers basedon whether the vehicle is located on a highway or a surface street. Inaddition or alternatively, the third computing system is configured todetermine the second list of possible maneuvers based on which of theplurality of sensors is functioning. In addition or alternatively, thethird computing system is configured to determine the second list ofpossible maneuvers based on available sensor functionality for theplurality of sensors.

In another example, the third computing system is further configured toprevent the vehicle from making certain types of maneuvers which areallowed by the first computing system. In this example, the certaintypes of maneuvers include turns of a certain curvature. In anotherexample, the third computing system is further configured to determinethat the vehicle is located on a highway or a surface street byreferring to pre-stored map information, and to generate and send thetrajectories further based on the determination that the vehicle islocated on a highway or surface street. In this example, the firstcomputing system is configured to detect traffic lights using trafficlight detection functionality and when the vehicle is determined to belocated on a highway, the third computing system is further configuredto generate trajectories without using the traffic light detectionfunctionality. In addition or alternatively, when the vehicle isdetermined to be located on a highway, the third computing system isfurther configured to generate trajectories in order to control thevehicle to exit the highway and reach a surface street. In this example,the third computing system is configured to search for an exit proximateto a surface street with certain characteristics, and to generatetrajectories in order to control the vehicle to exit the highway at theexit and reach a surface street. Alternatively, the first computingsystem is configured to detect traffic lights using traffic lightdetection functionality and when the vehicle is determined to be locatedon a surface street, the third computing system is further configured togenerate trajectories using the traffic light detection functionality.In addition or alternatively, the third computing system is furtherconfigured to generate trajectories in order to control the vehicle totravel at a predetermined speed when the vehicle is determined to belocated on a surface street, the predetermined speed providing for timeto detect and respond to objects in the environment of the vehicle. Asanother example, the system also includes the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle in accordance withan exemplary embodiment.

FIG. 2 is a functional diagram of aspects of the system of FIG. 1 inaccordance with aspects of the disclosure.

FIG. 3 is an example external view of a vehicle in accordance withaspects of the disclosure.

FIG. 4 is an example of map information in accordance with aspects ofthe disclosure.

FIG. 5 is an example representation of computing systems and messages inaccordance with aspects of the disclosure.

FIG. 6 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 7 is an example of a vehicle being maneuvered on a section ofroadway in accordance with aspects of the disclosure.

DETAILED DESCRIPTION

Overview

The technology relates to fallback driving behaviors for autonomousvehicles. During typical operation, a first computer system may generatea trajectory and send it to a second computing system in order tocontrol the vehicle according to that trajectory. The trajectoryincludes at least some portion that allows the vehicle to proceedtowards its end goal or destination, and thereafter, the trajectoryprovides fallback instructions for the vehicle to safely pull over,stop, etc. such that if the first computer system is not able togenerate a new trajectory because of some problem with the vehicle'svarious systems, the vehicle can safely pull over, stop, etc. This canbe problematic on highways and in situations where there is heavyvehicular or pedestrian traffic, and pulling over or stopping are notthe best actions for the vehicle to execute in the event of a failure ofone or more systems on the vehicle. To avoid this, if a given type offailure in some hardware or software system of the vehicle is detected,a third computer system, or a fall back system, with reduced performancemay be used to control the vehicle. Depending upon the driving situationand functionality available, the fallback system may behave in differentways with the overall goal of minimizing the likelihood of the vehiclestopping in its lane.

The first computing system may include a highly sophisticated plannersystem, perception system, and software stacks. For instance, perceptionsystem of the first computing system may include different softwaremodules designed to allow for the detection of different objects in thevehicle's environment. The software stack of the first computing systemmay use behavior models to predict how these objects are likely tobehave for some period of time into the future. Similarly, the plannersystem of the first computing system may use these predictions anddetailed map information in order to generate trajectories which allowthe vehicle to conduct all types of maneuvers.

The second computing system may be configured to receive trajectoriesfrom the first computing system and control various actuators of thevehicle in order to control the vehicle according to those receivedtrajectories. In this regard, the second computing system need notinclude planner or perception modules.

The third computing system may have different capabilities than thefirst and second computing systems. For instance, the third computingsystem may have greater computing capabilities than the second computer.At the same time, the third computing system may have less capabilitiesand power requirements than the first computing system. The thirdcomputing system may have a more streamlined version of the perceptionsystem and behavior models of the first computing system.

When an error in the first computing system is detected, the thirdcomputing system may take over for the first computing system in orderto generate trajectories for the second computing system. In order to doso, when an error in the first computing system is detected, the thirdcomputing system may determine a type of road, for instance highway orsurface street, on which the vehicle is driving by comparing thevehicle's last known or current location to map information. Based onthis comparison, the third computing system may determine how to controlthe vehicle and according to what functionality.

When an error in the first computing system is detected, the thirdcomputing system may assess the functionality of available sensors ofthe vehicle in order to determine what, if any, types of location thevehicle should avoid. In other words, there may be a hierarchy of roadscopes requiring different types of functionality, and if the vehicledoes not have the functionality required for a certain road scope, thethird computing system may avoid areas having that road scope.

The features described herein may allow the vehicle to continue to becontrolled safely in the event of various types of failures of a firstor primary computing system. In addition, these features allow fordifferent fallback behaviors depending upon what functionality isavailable and where the vehicle is currently located. In other words,the third computing system is able to leverage current functionality toget to the best possible outcome.

Example Systems

As shown in FIG. 1 , a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, buses, recreational vehicles,etc. The vehicle may have a plurality of computing systems, such asfirst computing systems 110, second computing system 120, and thirdcomputing system 130, each including one or more computing devices 112,122, 132. Together, these computing systems and devices may function asan autonomous driving computing system incorporated into vehicle 100.

Turning to FIG. 2 , which provides additional details of the first,second and third computing systems, each of these computing devices 112,122, 132 may include one or more processors 220, 221, 223, memory 230,233, 236 and other components typically present in general purposecomputing devices. The memory 230, 233, 236 stores informationaccessible by the one or more processors 220, 221, 222, includinginstructions 231, 234, 237 and data 232, 235, 238 that may be executedor otherwise used by the processor 220, 221, 222, respectively. Thememory 230, 233, 236 may be of any type capable of storing informationaccessible by the processor, including a computing device-readablemedium, or other medium that stores data that may be read with the aidof an electronic device, such as a hard-drive, memory card, ROM, RAM,DVD or other optical disks, as well as other write-capable and read-onlymemories. Systems and methods may include different combinations of theforegoing, whereby different portions of the instructions and data arestored on different types of media.

The instructions 231, 234, 237 may be any set of instructions to beexecuted directly (such as machine code) or indirectly (such as scripts)by the processor. For example, the instructions may be stored ascomputing device code on the computing device-readable medium. In thatregard, the terms “instructions” and “programs” and “software” may beused interchangeably herein. For instance, as discussed in furtherdetail below, the instructions may include various software stacks withvarious capabilities and functionalities. The instructions may be storedin object code format for direct processing by the processor, or in anyother computing device language including scripts or collections ofindependent source code modules that are interpreted on demand orcompiled in advance. Functions, methods and routines of the instructionsare explained in more detail below.

The data 232 may be retrieved, stored or modified by processor 220 inaccordance with the instructions 234. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The one or more processors 220, 221, 222 may be any conventionalprocessors, such as commercially available CPUs or GPUs. Alternatively,the one or more processors may be a dedicated device such as an ASIC orother hardware-based processor. Although FIG. 2 functionally illustratesthe processor, memory, and other elements of computing device 210 asbeing within the same block, it will be understood by those of ordinaryskill in the art that the processor, computing device, or memory mayactually include multiple processors, computing devices, or memoriesthat may or may not be stored within the same physical housing. Forexample, memory 230 of computing device 112 may be a hard drive or otherstorage media located in a housing different from that of computingdevice 112. Accordingly, references to a processor or computing devicewill be understood to include references to a collection of processorsor computing devices or memories that may or may not operate inparallel.

Each of the computing devices 112, 122, 132 may also include one or morewireless network connections 240, 241, 242 to facilitate communicationwith other computing devices, such as the client computing devices andserver computing devices described in detail below. The wireless networkconnections may include short range communication protocols such asBluetooth, Bluetooth low energy (LE), cellular connections, as well asvarious configurations and protocols including the Internet, World WideWeb, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing.

Computing device 122 of computing system 120 may also include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as one or more userinputs 243 (e.g., a mouse, keyboard, touch screen and/or microphone) andvarious electronic displays (e.g., a monitor having a screen or anyother electrical device that is operable to display information). Inthis example, the vehicle includes an internal electronic display 245 aswell as one or more speakers 244 to provide information or audio-visualexperiences. In this regard, internal electronic display 246 may belocated within a cabin of vehicle 100 and may be used by computingdevice 122 to provide information to passengers within the vehicle 100.

The computing system 110 may also include positioning system 250,perception system 260, and planner system 270. Each of these systems mayinclude sophisticated software modules and/or may include one or morededicated computing devices having processors and memory configured thesame or similarly to computing devices 112, 122, 132, processors 220,and memory 230. For instance, the positioning system 250 may alsoinclude other devices in communication with computing devices 122, suchas an accelerometer, gyroscope or another direction/speed detectiondevice to determine the direction and speed of the vehicle or changesthereto. By way of example only, an acceleration device may determineits pitch, yaw or roll (or changes thereto) relative to the direction ofgravity or a plane perpendicular thereto. The device may also trackincreases or decreases in speed and the direction of such changes. Thedevice's provision of location and orientation data as set forth hereinmay be provided automatically to the computing device 122, othercomputing devices and combinations of the foregoing.

The perception system 260 may include one or more components fordetecting objects external to the vehicle such as other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 260 may include lasers, sonar, radar,cameras and/or any other detection devices that record data which may beprocessed by computing device 110. In the case where the vehicle is apassenger vehicle such as a minivan, the minivan may include a laser orother sensors mounted on the roof or other convenient location. Forinstance, FIG. 3 is an example external view of vehicle 100. In thisexample, roof-top housing 310 and dome housing 312 may include a LIDARsensor or system as well as various cameras and radar units. Inaddition, housing 320 located at the front end of vehicle 100 andhousings 330, 332 on the driver's and passenger's sides of the vehiclemay each store a LIDAR sensor or system. For example, housing 330 islocated in front of driver door 360. Vehicle 100 also includes housings340, 342 for radar units and/or cameras also located on the roof ofvehicle 100. Additional radar units and cameras (not shown) may belocated at the front and rear ends of vehicle 100 and/or on otherpositions along the roof or roof-top housing 310.

The perception system 260 may include different software modulesdesigned to allow for the detection of different objects in thevehicle's environment. For example, the perception system of the firstcomputing system may include modules for detecting traffic lights andtheir states as well as other objects including different types of roadusers, such as pedestrians, vehicles, bicyclists, etc., as well as theircharacteristics such as location, orientation, heading, speed,acceleration, etc. The software stack of the first computing system mayuse sophisticated behavior models to predict how these objects arelikely to behave for some period of time into the future. Similarly, theplanner system of the first computing system may use these predictionsand detailed map information in order to generate trajectories whichallow the vehicle to conduct all types of maneuvers.

The planner system 270 may be used by computing device 112 in order tofollow a route to a location. In this regard, the planner system 270and/or data 132 of computing devices 112 may store detailed mapinformation, e.g., highly detailed maps identifying the shape andelevation of roadways, lane lines, intersections, crosswalks, speedlimits, traffic signals, buildings, signs, real time trafficinformation, pull over spots, vegetation, or other such objects,features and information.

FIG. 4 is an example of map information 400 for a section of roadwayincluding highway 402 and surface street 404. In this example, highway402 passes over surface street 404. The map information 400 may be alocal version of the map information stored in the memory 230 of thecomputing devices 112. Other versions of the map information may also bestored in the memory of the computing devices 132 as discussed furtherbelow. In this example, the map information 400 includes informationidentifying the shape, location, and other characteristics of lane lines410, 412, 414, highway exit ramp 420, highway entrance ramp 422,shoulder areas 430, 432, etc.

The map information may also define discrete portions of lanes ofhighway 402 and surface street 404 as individual segments 440 (shownconnected end to end by dots) of a few meters or more or less, shownonly with respect to a single lane of highway 402 for clarity. Ofcourse, each lane of highway 402, surface street 404, highway exit ramp420 and highway entrance ramp 422 may each be defined as road segments.

The map information may also differentiate between types of drivablesurfaces, such as between highway and surface streets, such asresidential streets, state or county roads, or any other non-highwaystreet. In addition or alternatively, these areas may also be flagged ashigh pedestrian and/or high vehicular traffic areas. In this regard, thearea and/or road segments of highway 402 may be designated as a highway,and the area and/or road segments of surface street 404, may bedesignated as a surface street.

Although, the map information is depicted herein as an image-based map,the map information need not be entirely image based (for example,raster). For example, the map information may include one or moreroadgraphs or graph networks of information such as roads, lanes,intersections, and the connections between these features which may berepresented by road segments. Each feature may be stored as graph dataand may be associated with information such as a geographic location andwhether or not it is linked to other related features, for example, astop sign may be linked to a road and an intersection, etc. In someexamples, the associated data may include grid-based indices of aroadgraph to allow for efficient lookup of certain roadgraph features.

The various systems of the computing system 120 may function in order todetermine how to control the vehicle 100. As an example, a perceptionsystem software module of the perception system 260 may use sensor datagenerated by the one or more sensors to detect and identify objects andtheir characteristics. These characteristics may include location,object type, heading, orientation, speed, acceleration, change inacceleration, size, shape, etc. In some instances, characteristics maybe used by the computing devices 112 to determine a predicted futurebehavior for a detected object.

In one example, computing devices 112 may be operable to predict anothervehicle's future movement based solely on the other vehicle's instantdirection, acceleration/deceleration and velocity, e.g., that the othervehicle's current direction and movement will continue. However, memory230 of computing devices 112 may also store behavior models thatprovides the probability of one or more actions being taken a detectedobject. To increase the usefulness of these behavior models, eachbehavior model may be associated with a specific type of object. Forinstance, one type of behavior model may be used for objects identifiedas pedestrians, another type of behavior model may be used for objectsidentified as vehicles, another type of behavior may be used for objectsidentified as bicycles or bicyclists, etc. The behavior models may beused by the computing devices 112 in order to predict a detected objectfuture movement by analyzing data relating to the other vehicle'scurrent surroundings (such as the detected or estimated size, shape,location, orientation, heading, velocity, acceleration or deceleration,change in acceleration or deceleration, etc.), and determining how thatother object will likely respond to those surroundings. In this regard,the behavior models may function from an object-centric, view of theobject's environment, in that the system determines what the otherobjects are perceiving in order to better predict how those objects willbehave. In this regard, in at least some instances, the behavior modelsmay also indicate whether the predicted behavior for an object isresponsive to a particular other object including the vehicle 100.

In other instances, the characteristics may be put into one or moredetection system software modules stored in memory 230 of computingdevices 112. These detection system software modules may include, forinstance, a traffic light detection system software module configured todetect the states of known traffic signals, construction zone detectionsystem software module configured to detect construction zones fromsensor data generated by the one or more sensors of the vehicle as wellas an emergency vehicle detection system configured to detect emergencyvehicles from sensor data generated by sensors of the vehicle. Each ofthese detection system software modules may uses various models tooutput a likelihood of a construction zone or an object being anemergency vehicle.

Detected objects, predicted future behaviors, various likelihoods fromdetection system software modules, the map information identifying thevehicle's environment, position information from the positioning system250 identifying the location and orientation of the vehicle, and adestination for the vehicle as well as feedback from various othersystems of the vehicle may be input into a planner system softwaremodule of the planner system 270. The planner system 270 and/orcomputing devices 112 may use this input to generate a route andtrajectories for the vehicle to follow for some brief period of timeinto the future. These trajectories may be generated periodically, forinstance, 10 times per second or more or less, and may extend for sometime and distance into the future in order to allow the vehicle followthe route to the destination. These trajectories may be generated as“desired paths” in order to avoid obstacles, obey laws and generallydrive safely and effectively. Each trajectory may define variousrequirements for the vehicle's acceleration, speed, and position atdifferent times along the trajectory. Each trajectory may include afirst portion designed to cause the vehicle to reach a destination orend goal and a second portion designed to allow the vehicle to pull overor stop safely. In this regard, if a new trajectory is not received intime, the vehicle can safely pull over.

The second computing system 120 may be configured to receivetrajectories from the first computing system and control variousactuators of the vehicle in order to control the vehicle according tothose received trajectories. For instance, a control system softwaremodule stored in the memory 232 of the computing devices 122 may beconfigured to control movement of the vehicle, for instance bycontrolling braking, acceleration and steering of the vehicle, in orderto follow a trajectory. As an example, computing devices 122 mayinteract with one or more actuators of the vehicle's deceleration system140 and/or acceleration system 150, such as brake pedal or other input,accelerator pedal or other input, and/or a power system 160 of thevehicle in order to control the speed of the vehicle. Similarly,steering system 170 may be used by the computing devices 122 in order tocontrol the direction of vehicle 100. For example, if vehicle 100 isconfigured for use on a road, such as a car or truck, the steeringsystem may include components to control the angle of wheels to turn thevehicle. Signaling system 180 may be used by computing device 110 inorder to signal the vehicle's intent to other drivers or vehicles, forexample, by lighting turn signals or brake lights when needed.

The power system 160 may include various features configured to supplypower to the first, second, and third computing systems as well as othersystems of the vehicle. In this regard, the power system 160 may includeone or more batteries as well as an electric motor and/or a gasoline ordiesel-powered engine.

The second computing system 120 need not include the aforementionedplanner or perception hardware or software modules of the firstcomputing systems. However, in some instances, the second computingsystem may include fairly simplistic detection systems, such as aforward-facing radar unit, configured to cause the second computingdevice to activate emergency braking in the event the second computingdevice is following the second portion of a trajectory and an objectappears directly in front of the vehicle.

The third computing system 130 may have different capabilities than thefirst and second computing systems. For instance, the third computingsystem 130 may have greater computing capabilities than the secondcomputer. At the same time, the third computing system may also haveless capabilities and power requirements than the first computingsystem.

The third computing system 130 may also have a more streamlined versionof some of the systems of the first computing system 110. For instance,the third computing system may include a copy of the map information inmemory 232 of computing device 122 a positioning system 290, aperception system 292, and a planner system 294. Alternatively, ratherthan a copy, the third computing system may access the map informationdirectly from memory 230 of computing device 112. The positioning system290 may be the same or configured the same as or similarly topositioning system 250. However, each of the perception system 292 andthe planner system 294 may be configured with different functionalitythan those of the first computing system 110 that results in lesscomputing, resource, and/or other resource requirements for the thirdcomputing system 130 as compared to the first computing system 110.

For instance, the perception system 292 of the third computing system130 may be configured to detect objects using, for instance, the sensorsof the perception system 260 and/or additional sensors (for instancecameras, radar units, sonar, LIDAR sensors, etc.) specific to theperception system 292 itself. However, all objects that do not appear inthe map information of the computing device 122, or rather, other roadusers, such as pedestrians, bicyclists, and vehicles, may all beresponded to as if they were the same type of object. In this regard,the perception system of the third computing system need notdifferentiate different types of objects. As such, the third computingsystem 130 may perform more streamlined predictions of how objects willbehave by making less dense (fewer estimates in the same period of time)and less precise predictions. In this regard, the third computing devicemay not include the aforementioned sophisticated behavior predictionsoftware modules. Instead, the third computing device 132 may use fairlysimplistic behavior models, such as simple extrapolating current speed,acceleration, deceleration, orientation, change in acceleration, changein orientation, etc. for a few seconds in the future or simply assumingthat the vehicle will continue at its current speed, acceleration,deceleration, etc. while following a lane in which the object iscurrently located. As an example, all objects on a highway may beassumed by the third computing device to either continue to follow theirlane at current speed, etc.

As another example, the planner system 294 of the third computing systemmay generate trajectories for the vehicle to follow. As with plannersystem 270, detected objects, predicted future behaviors, variouslikelihoods from detection system software modules, the map informationidentifying the vehicle's environment, position information from thepositioning system 250 identifying the location and orientation of thevehicle, and a destination for the vehicle as well as feedback fromvarious other systems of the vehicle may be input into a planner systemsoftware module of the planner system 294. The planner system 294 and/orcomputing devices 112 may use this input to generate a route andtrajectories for the vehicle to follow for some brief period of timeinto the future. These trajectories may be generated periodically, forinstance, 10 times per second or more or less, and may extend for sometime and distance into the future in order to allow the vehicle followthe route to the destination. These trajectories may be generated as“desired paths” in order to avoid obstacles, obey laws and generallydrive safely and effectively. Each trajectory may define variousrequirements for the vehicle's acceleration, speed, and position atdifferent times along the trajectory. Each trajectory may include afirst portion designed to cause the vehicle to reach a destination orend goal and a second portion designed to allow the vehicle to pull overor stop safely. In this regard, if a new trajectory is not received intime, the vehicle can safely pull over.

However, the planner system 294 may be limited in the types of maneuversthat its generated trajectories may include. In some instances, thetypes of maneuvers may be defined as one or more predetermined lists ofmaneuvers. Which predetermined list of maneuvers used by the plannersystem 294 may be determined based on a type of failures of the firstcomputing system as well as where the vehicle is located as discussedfurther below. For example, the third computing system may only be ableto generate trajectories for the vehicle to perform turns of certaincurvature, turns of a certain type, etc. As another instance, the typesof reactions necessary on a highway may be different from those that arenecessary on surface streets, and thus, allowed maneuvers may bedetermined based on the location of the vehicle.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

As noted above, in order to control vehicle 100 in the autonomousdriving mode, the first computing system 110, second computing system120, and third computing system 130 may exchange different types ofmessages and information with one another. This information may be sent,for instance, via a Controller Area Network (CAN) bus of the vehicle.Referring to FIG. 5 , the computing devices 112 of the first computingsystem 110 may send trajectory messages 510 including trajectories anderror messages 520 including errors to the computing devices 122 of thesecond computing system 120. In addition, these messages, and inparticular the error messages 520 may be received by the computingdevices 132 of the third computing system 130. The computing devices 132of the third computing system 130 may also send trajectory messages 530to the computing devices 122 of the second computing system 120. In someinstances, the computing devices 132 of the third computing system 130may also send command messages 540 and/or command messages 550 to thecomputing devices 112 of the first computing system 110 and to thecomputing devices 122 of the second computing system 120.

FIG. 6 is an example flow diagram 600 for controlling a vehicle, such asvehicle 100, in an autonomous driving mode using first, second and thirdcomputing systems such as first computing system 110, second computingsystem 120, and third computing system 130. For instance at block 602,the planner system 270 of the first computing system may use variousinputs to generate new trajectories for the vehicle as discussed above.

As shown in blocks 604 and 606, these trajectories may be sent by thecomputing devices 112 of the computing system 110 to the computingdevices 122 of the computing system 120, for instance, via trajectorymessages 510 of FIG. 5 . In response, as shown in block 608, thecomputing devices 122 may control the vehicle in the autonomous drivingmode according to the received trajectory. For instance as noted above,the computing devices 122 may control the vehicle to follow thetrajectory by sending commands to control the actuators of thedeceleration system 140, acceleration system 150, steering system 170,and/or power system 160.

FIG. 7 provides an example of vehicle 100 functioning according toblocks 602-608 of flow diagram 600. In this example, vehicle 100 isbeing maneuvered on a section of roadway 700 including highway 702 andsurface street 704. In example of FIG. 7 , highway 702 and surfacestreet 704 correspond to highway 402 and surface street 704 of the mapinformation 400, respectively. In this example, lane lines 710, 712, and714 correspond to the shape, location, and other characteristics of lanelines 410, 412, and 414, respectively. Similarly, highway exit ramp 720,highway entrance ramp 722, shoulder areas 730, 732, etc. correspond tothe shape, location, and other characteristics of highway exit ramp 420,highway entrance ramp 422, shoulder areas 430, 432, etc. Vehicle 100 isapproaching highway entrance ramp 722 from surface street 704, andfollowing a trajectory 770. At this point, trajectory 770 is a currenttrajectory having a first portion 780 to control the vehicle to adestination and a second portion 790 to control the vehicle to pull overand stop in the event the computing system 110 does not provide a newtrajectory the second computing system 120.

As such, the vehicle 100 will proceed along the trajectory, following atleast the first portion of the current trajectory, such as first portion780, until a new trajectory is received by the computing devices 122 ofthe second computing system 120 from the planner system 270. Typically,a new trajectory would be generated, received by the computing devices122, and acted upon prior to the start of the second portion of acurrent trajectory, such as second portion 790. Of course, if not, thecomputing devices 110 will seamlessly continue to control the vehicle inorder to follow the second portion of the trajectory and cause thevehicle to pull over, stop, etc.

Returning to FIG. 6 , while this process is occurring, as shown in block610, the computing devices 132 of the third computing system 130 maymonitor the messages sent from the first computing system to the secondcomputing system, including the aforementioned error messages 520. Theseerror messages may indicate errors in the first computing system. Inthis regard, the third computing system may determine whether an erroris detected as shown in block 612. If no errors are detected, the thirdcomputing system may continue to monitor the messages.

When an error in the first computing system is detected, the thirdcomputing system may take over for the first computing system in orderto generate trajectories and send these trajectories to the secondcomputing system. In order to do so, the third computing system mayrequest for first computing system to be shut down, for instance bysending a command message, such as command messages 540 of FIG. 5 , tothe first computing system. Alternatively, both the first and thirdcomputing system may continue to simultaneously generating trajectories(with some amount of common initial smoothness) all of the time. In thisinstance, the second computing system may arbitrate between thetrajectories from each of the first and third computing system anddetermine which to use. To avoid the need for this arbitration, thethird computing system may send a command message, such as commandmessages 550, to the second computing system to disregard trajectoriesfrom first computing system.

In order to do so, when an error in the first computing system isdetected, the computing devices 132 of the third computing system 130may determine a type of road on which the vehicle is driving as shown inblock 614. This may involve comparing the vehicle's last known orcurrent location from the positioning system 250 and/or positioningsystem 290 to map information stored in memory 236 of computing devices132. As an example, using the location and the map information, thecomputing devices 132 may determine a road segment on which the vehicleis traveling. This road segment may be associated with an identifierindicating a type of the road segment, for instance, a highway or asurface street. Thus, put another way, the computing devices 132 maydetermine whether the vehicle is on a highway or a surface street.

As shown in block 616, based on the determined type of road, the thirdcomputing system may determine how to generate trajectories to controlthe vehicle and according to what functionality. For instance, if thevehicle is determined to be located on a highway, in some situations,the third computing system need not use certain features of theperception system 292 such as the ability to detect traffic lights andtraffic lights states using the aforementioned traffic light detectionsystem software module or functionality. In this regard, thisfunctionality may be turned “off” and need not be used by the thirdcomputing system to generate trajectories. In addition, detectingobjects very close to the vehicle may be less critical, and may be givenlower processing priority, as there are not likely to be pedestrians orbicyclists on highways. Still further, in situations where there is notmuch traffic, because the vehicle is more likely to be continuouslymoving, it may be unlikely that another road user is going to enter intothe vehicle's blind spot, and again detecting objects very close to thevehicle may be given lower processing priority.

These trajectories may include those that allow the vehicle to maintainits lane or move, one lane at a time, to the right such that the vehiclebecomes in a position to exit the highway in order to reach surfacestreets. In addition, when generating trajectories, the third computingsystem may search for a nearby exit proximate to surface streets withcertain characteristics, such as available parking, routes that avoidtraffic signals, routes that avoid specific maneuvers (such asunprotected turns), routes that avoid known construction zones, etc.

In addition or alternatively, when generating trajectories, the thirdcomputing system may also consider different aspects of the thirdcomputing system's functionality. For instance, when the vehicleoperating on highways or surface streets the perception system 292 andplanner system 294 of the third computing system 130 need not actuallydifferentiate between different types of object. In addition, thecomputing devices 132 of the third computing system 130 may make verybasic predictions about these objects. However, in some instances, somedifferentiation may be performed automatically by software modules ofthe sensors themselves, such as a LIDAR system, such as that of domehousing 312, which can identify and differentiate between pedestriansand vehicles automatically. Of course, these objects may all still betreated the same in order to limit the processing and power requirementsof the third computing system 130.

In some instances, when an error in the first computing system 110 isdetected, the third computing system 130 may also assess thefunctionality of available sensors of the vehicle in order to determinewhat if any types of location the vehicle should avoid. at is one of theprimary sensors for that use case. In other words, there may be ahierarchy of road scopes requiring different types of functionality, andif the vehicle does not have the functionality required for a certainroad scope, the third computing system may avoid areas having that roadscope. For instance, certain types of failures, such as power system orindividual sensor problems, may reduce the vehicle's capability ofdetecting objects in the vehicle's environment at certain locations. Inthis regard, the third computing system 130 may determine which of thevehicle's sensors are still functional and generate trajectoriesaccordingly. For example, if a rear-left corner radar of the vehicle 100has been compromised (i.e. is no longer functioning to operatingspecifications), then lane changes to the left on high speed roads maybe avoided, since the rear-left corner radar may be important for suchmaneuvers. As another example, in some instances, such as where thethird computing system is not able to detect traffic lights due to anerror in a camera sensor used for this purpose, when the vehicle is on ahighway, the third computing system may cause the vehicle to stay on ahighway and pull over on the highway rather than entering surfacestreets. In addition, the third computing system may avoid routes withcertain types of maneuvers or those that pass through certain types ofareas. This may include routes with unprotected turns, multipoint turnsincluding those where there is or is not a shoulder area, high speedlane changes, parking lots, construction zones, school zones, 4-waystops, narrow roads, one-way roads, low-friction roads (i.e. gravel),roads with poor lighting at night, roads with suicide lanes, etc.

Based on the determination of how to generate trajectories and accordingto what functionality, the computing devices 132 of the third computingsystem 130 may generate trajectories at block 618 as described above. Asshown in block 620 and 606, these trajectories may be sent to andreceived by the computing devices 122 of computing system 120, forinstance, via the trajectory messages 530 of FIG. 5 . In response, asshown in block 608, the computing devices 112 may control the vehicle inthe autonomous driving mode according to the received trajectory. Forinstance as noted above, the computing devices 112 may control thevehicle to follow the trajectory by sending commands to control theactuators of the deceleration system 160, acceleration system 150,steering system 170, and/or power system 160.

Returning to block 614, when the vehicle is determined to be located onsurface streets or once the vehicle reaches surface streets, the thirdcomputing system may require the functionality for detecting trafficlights and traffic lights states. In addition, detecting objects veryclose to the vehicle, for instance, alongside of the vehicle, may bemuch more important. Because of this, the third computing system 130 maygenerate trajectories which cause the vehicle to be controlled in orderto drive at a predetermined limited speed, such as 10 miles per hour orless, to increase the likelihood of such objects being detected byfunctioning sensors of the vehicle and to provide the third computingsystem with additional time to detect and respond to these objects. Inaddition, as noted when driving on surface streets, the planner system294 of the third computing system 130 may only allow or generatetrajectories for certain types of maneuvers and may prohibit or limitgenerating trajectories for other types of maneuvers such as turns withcertain types of curvature, unprotected turns, multipoint turns, etc. Insome examples, when driving on surface streets, the third computingsystem may also generate trajectories in order to prevent the vehiclefrom entering certain areas, such as those identified in the mapinformation as high pedestrian and/or vehicular traffic areas. In thisregard, the planner system of the third computing system may avoidgenerating trajectories that route the vehicle through these areas.

Again, based on the determination of how to generate trajectories andaccording to what functionality, the third computing devices maygenerate trajectories at block 618 as described above. As shown inblocks 620 and 606, these trajectories may be sent to and received bythe computing devices 122 of the computing system 120, for instance, viathe trajectory messages 530 of FIG. 5 . In response, as shown in block608, the computing devices 112 may control the vehicle in the autonomousdriving mode according to the received trajectory. For instance as notedabove, the computing devices 112 may control the vehicle to follow thetrajectory by sending commands to control the actuators of thedeceleration system 140, acceleration system 150, steering system 170,and/or power system 160.

The features described herein may allow the vehicle to continue to becontrolled safely in the event of various types of failures of a firstor primary computing system. In addition, these features allow fordifferent fallback behaviors depending upon what functionality isavailable and where the vehicle is currently located. In other words,the third computing system is able to leverage current functionality toget to the best possible outcome. For instance, during highway driving,the vehicle may continue driving in its current lane before attemptingto change lanes or pull over as the computational power and sensorsrequired to accomplish this are significantly less than what would berequired to accomplish the same for non-highway driving (i.e. surfacestreets). When driving on surface streets, the vehicle may be sloweddown dramatically, for instance to 10 mph or more or less, in order tobetter enable the vehicle's sensors and other systems to detect andrespond to objects. This additional level of sophistication over thetypical, severe reaction of stopping a vehicle immediately may besignificantly safer especially in certain driving situations such ashighway driving and/or high traffic areas.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

The invention claimed is:
 1. A control system for controlling a vehiclein an autonomous driving mode, the control system comprising: aplurality of sensors configured to generate sensor data; a primarycomputing system configured to: generate different behavior predictionsbased on each of different types of road users in the sensor data,generate trajectories using the sensor data, and send the generatedtrajectories to a secondary computing system configured to cause thevehicle to follow a received trajectory; and a backup computing systemconfigured to, when there is a failure of the primary computing system,generate and send one or more trajectories to the secondary computingsystem to control the vehicle to maneuver to a stopping location.
 2. Thecontrol system of claim 1, wherein the backup computing system sends theone or more trajectories to the secondary computing system based on atype of road on which the vehicle is currently traveling.
 3. The controlsystem of claim 2, wherein the type of road is a highway or a surfacestreet.
 4. The control system of claim 1, wherein the backup computingsystem has greater computing capabilities than the secondary computingsystem.
 5. The control system of claim 1, wherein the backup computingsystem has lesser computing capabilities than the primary computingsystem.
 6. The control system of claim 1, wherein the backup computingsystem is further configured to prevent the vehicle from making certaintypes of maneuvers which are allowed by the primary computing system. 7.The control system of claim 6, wherein the certain types of maneuversinclude turns of a certain curvature.
 8. The control system of claim 1,wherein the failure relates to one or more of the plurality of sensors,and the backup computing system is configured to generate the one ormore trajectories further based on which of the plurality of sensors isfunctioning.
 9. The control system of claim 1, wherein the secondarycomputing system is not configured to differentiate between thedifferent types of road users in the sensor data.
 10. The control systemof claim 1, wherein the backup computing system is further configured togenerate the different behavior predictions in a same way as the primarycomputing system.
 11. The control system of claim 1, wherein the backupcomputing system is further configured to generate behavior predictionscorresponding to either following a current lane or changing lanes inresponse to a detection of any given road user on a highway.
 12. Thecontrol system of claim 1, wherein the primary computing system isfurther configured to generate trajectories according to a first list ofpossible maneuvers, and wherein the backup computing system is furtherconfigured to generate the one or more trajectories according to asecond list of possible maneuvers that is smaller than the first list ofpossible maneuvers.
 13. The control system of claim 12, wherein thebackup computing system is further configured to determine the secondlist of possible maneuvers based on whether the vehicle is located on ahighway or a surface street.
 14. The control system of claim 12, whereinthe backup computing system is further configured to determine thesecond list of possible maneuvers based on which of the plurality ofsensors is functioning.
 15. An autonomous vehicle comprising: aplurality of sensors configured to generate sensor data; a primarycomputing system configured to generate different behavior predictionsbased on each of different types of road users in the sensor data, andgenerate trajectories using the sensor data; a secondary computingsystem configured to receive the generated trajectories from the primarycomputing system to cause the autonomous vehicle to follow a receivedtrajectory; and a backup computing system configured to, when there is afailure of the primary computing system, generate and send one or moretrajectories to the secondary computing system to control the autonomousvehicle to maneuver to a stopping location.
 16. The autonomous vehicleof claim 15, wherein the backup computing system sends the one or moretrajectories to the secondary computing system based on a type of roadon which the autonomous vehicle is currently traveling.
 17. Theautonomous vehicle of claim 16, wherein the type of road is a highway ora surface street.
 18. A control system fir controlling a vehicle in anautonomous driving mode, the control system comprising: a plurality ofsensors configured to generate sensor data; a primary computing systemconfigured to generate different behavior predictions based on each ofdifferent types of road users in the sensor data, and generatetrajectories using the sensor data; a secondary computing systemconfigured to receive the generated trajectories from the primarycomputing system to cause the vehicle to follow a received trajectory;and a backup computing system configured to, when there is a failure ofthe primary computing system, generate and send one or more trajectoriesto the secondary computing system to control the vehicle to maneuver toa stopping location.
 19. The control system of claim 18, wherein thebackup computing system sends the one or more trajectories to thesecondary computing system based on a type of road on which the vehicleis currently traveling.
 20. The control system of claim 19, wherein thetype of road is a highway or a surface street.